{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "pyspark 命令及其常用的参数如下:\n",
    "\n",
    "pyspark --master < master-url >\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Spark 的运行模式取决于传递给 SparkContext 的 Master URL 的值。 Master\n",
    "URL 可以是以下任一种形式:\n",
    "* local 使用一个 Worker 线程本地化运行 SPARK( 完全不并行 )\n",
    "* local[*] 使用逻辑 CPU 个数数量的线程来本地化运行 Spark\n",
    "* local[K] 使用 K 个 Worker 线程本地化运行 Spark (理想情况下, K 应该根据运\n",
    "行机器的 CPU 核数设定)\n",
    "* spark://HOST:PORT 连接到指定的 Spark standalone master 。默认端口是\n",
    "7077\n",
    "* yarn-client 以客户端模式连接 YARN 集群。集群的位置可以在\n",
    "HADOOP_CONF_DIR 环境变量中找到\n",
    "* yarn-cluster 以集群模式连接 YARN 集群。集群的位置可以在\n",
    "HADOOP_CONF_DIR 环境变量中找到\n",
    "* mesos://HOST:PORT 连接到指定的 Mesos 集群。默认接口是 5050"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "在 Spark 中采用本地模式启动 pyspark 的命令主\n",
    "要包含以下参数:\n",
    "\n",
    "--master :这个参数表示当前的 pyspark 要连接\n",
    "到哪个 master ,如果是 local[*] ,就是使用本地\n",
    "模式启动 pyspark ,其中,中括号内的星号表示\n",
    "需要使用几个 CPU 核心 (core) ,也就是启动几个\n",
    "线程模拟 Spark 集群\n",
    "\n",
    "--jars : 这个参数用于把相关的 JAR 包添加到\n",
    "CLASSPATH 中;如果有多个 jar 包,可以使用逗\n",
    "号分隔符连接它们\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "比如,要采用本地模式,在 4 个 CPU 核心上运行 pyspark :\n",
    "\n",
    "$ cd /usr/local/spark\n",
    "\n",
    "$ ./bin/pyspark --master local[4]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "开发 Spark 独立应用程序"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}